Skip to content

feat(core): complete native local hybrid search - #1097

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-local-hybrid-search
Oct 11, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-local-hybrid-search

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Summary

Complete the Rust execution needed to move PyPaimon's existing local hybrid search into one native operation.

  • Apply accumulated data and partition predicates to each vector/full-text route before its own Top-K, reusing the existing route readers and their Java coverage/refinement semantics.
  • Resolve one snapshot for the operation and preserve the Snapshot object, including retained tags whose original snapshot file has expired. Accept an already resolved snapshot or an explicitly empty read view from the caller.
  • Execute at most four routes concurrently and retain route order for Java float fusion. Reuse the existing RRF, weighted-score and MRR rankers, route weights, route limits and final limit.
  • Use the full-text DSL API used by Java hybrid routes and report invalid configuration before an empty result, including unknown vector columns and zero final limits.
  • Add thin Python bindings and type stubs. The binding forwards configuration, predicates and snapshot JSON to core; planning, route execution and fusion remain in Rust core.
  • Reuse the DataFusion Arrow schema conversion when materializing hybrid results so text columns match the declared Utf8View output. Update SQL regressions to require an actual full-text definition, as Java does.

The implementation follows HybridSearchBuilderImpl, HybridSearchRanker and the existing Java vector/full-text route readers.

Verification

  • cargo test --locked -p paimon --lib --features fulltext hybrid_search_builder: 20 passed.
  • cargo test --locked -p paimon-datafusion --features fulltext --test read_tables hybrid_search: 5 passed.
  • cargo +1.98.0 clippy --locked -p paimon -p paimon-datafusion -p pypaimon_rust --all-targets --features fulltext,vortex -- -D warnings: passed.
  • cargo check --locked -p paimon --lib without fulltext: passed.
  • cargo fmt --all -- --check and git diff --check: passed.
  • Rebuilt the Python extension with maturin and ran the paired PyPaimon REST suite with native planning, reading, writing, updating and committing enabled: 73 passed, 3 subtests passed.

Regressions cover real vector/full-text/scalar indexes, partial coverage, filtering before Top-K, all three rankers, more routes than the worker bound, authoritative REST snapshots and errors, fixed empty views, retained tags, malformed DSL, unsupported literals and query authorization. Two independent review passes found and fixed the missing-column and caller-pinned-view issues.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the native local hybrid search implementation against the Java HybridSearchBuilder and ranker semantics. Snapshot pinning, per-route filtering before Top-K, address-space validation, route concurrency, score fusion, primary-key materialization, and DataFusion score projection are consistent. Validation passed with cargo fmt, the targeted paimon hybrid-search tests (20 passed), and the targeted DataFusion hybrid-search tests (4 passed). No blocking issues found.

@JingsongLi
JingsongLi merged commit a1bec7b into apache:main Oct 11, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants